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United  States  General  Accounting  Office  National  Security  and 

Washington,  D.C.  20548  International  Affairs  Division 


B-281196 
May  9,  2000 

The  Honorable  Christopher  Shays 
Chairman,  Subcommittee  on  National  Security, 

Veterans’  Affairs,  and  International  Relations 
Committee  on  Government  Reform 
House  of  Representatives 

Dear  Mr.  Chairman: 

The  Joint  Strike  Fighter  Program  is  intended  to  produce  an  affordable, 
next-generation  aircraft  to  replace  the  Department  of  Defenses  (DOD) 
aging  aircraft  inventory  The  first  aircraft  deliveries  are  scheduled  to  begin 
in  2008.  As  currently  planned,  the  program  will  cost  about  $200  billion  to 
develop  and  procure  over  3,000  aircraft  and  related  support  equipment  for 
the  Air  Force,  the  Marine  Corps,  the  Navy,  and  Great  Britain. 

DOD  has  designated  the  Joint  Strike  Fighter  Program  as  a  flagship  program 
for  acquisition.  To  date,  the  program  has  awarded  contracts  totaling  over 
$2  billion  to  Boeing  and  Lockheed  Martin  for  the  current  concept 
demonstration  phase.  Under  these  contracts,  both  contractors  will  build 
the  aircraft  they  plan  to  fly  in  the  demonstration  phase  and  also  design  the 
aircraft  they  plan  to  build  in  the  next  phase  of  the  development  program — 
engineering  and  manufacturing  development.  During  engineering  and 
manufacturing  development,  the  Joint  Strike  Fighter  will  be  fully 
developed,  engineered,  designed,  fabricated,  tested,  and  evaluated  to 
demonstrate  that  the  production  aircraft  will  meet  stated  requirements. 
DOD  is  scheduled  to  award  the  contract  for  engineering  and  manufacturing 
development  to  either  Boeing  or  Lockheed  Martin  in  April  2001. 1 

At  your  request,  we  reviewed  the  Joint  Strike  Fighter  Program  to 
(1)  provide  information  on  the  acquisition  strategy  and  (2)  to  determine 
whether  the  strategy  is  being  implemented  in  a  manner  that  will  ensure  that 
the  acquisition  strategy  objectives  will  be  achieved.  With  your  permission, 
we  discussed  a  draft  of  this  report  during  a  March  16,  2000,  joint  hearing  by 


1  At  the  time  of  this  report,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  was  reviewing  competition  and  industrial-base  implications  of  the  Joint  Strike 
Fighter  acquisition  strategy. 
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Results  in  Brief 


the  Subcommittees  on  Military  Procurement  and  on  Military  Research  and 
Development,  House  Committee  on  Armed  Services.  At  the  time  of  the 
hearing,  we  had  not  received  DOD’s  comments  on  our  report.  This  report 
contains  DOD’s  comments  and  our  evaluation  of  them. 


The  key  objective  of  the  Joint  Strike  Fighter  acquisition  strategy  is 
affordability — reducing  the  development,  production,  and  ownership  costs 
of  the  program  relative  to  prior  fighter  aircraft  programs.  DOD  expects  the 
Joint  Strike  Fighter  acquisition  strategy  to  save  nearly  $18  billion  (in  fiscal 
year  1995  dollars)  in  development  costs.  To  achieve  its  affordability 
objective,  the  Joint  Strike  Fighter  program  office  has  incorporated  various 
DOD  and  commercial  acquisition  initiatives  into  the  Joint  Strike  Fighter 
acquisition  strategy.  These  initiatives  include  modifying  the  traditional 
weapons  acquisition  cycle,  revising  the  requirements  determination 
process,  and  developing  critical  technologies  to  a  level  where  they 
represent  low  technical  risk  before  the  engineering  and  manufacturing 
contract  is  awarded.  The  expectation  is  that  incorporating  these  initiatives 
into  the  Joint  Strike  Fighter  acquisition  strategy  will  result  in  a  better 
match  between  the  maturity2  of  key  technologies  and  the  aircraft's 
requirements.  Matching  the  requirements  and  the  maturity  of  technology 
when  a  program  enters  engineering  and  manufacturing  development  is  a 
critical  determinant  of  a  program  s  success.  Once  the  development  phase 
begins,  a  large,  fixed  investment  in  the  form  of  human  capital,  facilities,  and 
materials  is  sunk  into  the  program  and  any  significant  changes  will  have  a 
large,  rippling  effect  on  cost  and  schedule.  Beginning  the  engineering  and 
manufacturing  development  phase  when  critical  technologies  are  at  a  low 
level  of  maturity  serves  to  significantly  increase  program  risk  and  the 
likelihood  of  schedule  delays,  which  in  turn  result  in  increased  program 
costs. 

The  Joint  Strike  Fighter  program  office’s  implementation  of  its  acquisition 
strategy  will  not  ensure  that  the  Joint  Strike  Fighter  program  will  enter  the 
engineering  and  manufacturing  development  phase  with  low  technical  risk. 
The  aircraft  being  produced  during  the  concept  demonstration  phase  are 
not  intended  to  demonstrate  many  of  the  technologies  considered  critical 
for  achieving  Joint  Strike  Fighter  program  cost  and  performance 
requirements.  Instead,  many  of  these  technologies — such  as  avionics,  flight 


2  A  technology  is  considered  to  be  mature  when  it  has  been  developed  to  a  point  that  it  can 
be  readily  integrated  into  a  new  product  and  counted  on  to  meet  product  requirements. 
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systems,  manufacturing  and  producibility,  propulsion,  supportability,  and 
weapons  delivery  system — will  only  be  demonstrated  in  laboratory  or 
ground-testing  environments.  Therefore,  these  critical  technologies  will  be 
at  low  levels  of  technical  maturity  when  the  engineering  and  manufacturing 
development  contract  is  scheduled  to  be  awarded.  In  addition,  when  the 
competing  contractors  experienced  design  problems  and  cost  overruns, 
DOD  restructured  the  program  in  a  manner  that  will  provide  less 
information  than  originally  planned  prior  to  selecting  between  the  two 
competing  contractors.  Specifically,  this  program  restructure  moves  away 
from  best  commercial  practices  that  were  evident  in  the  original  strategy, 
where  technology  was  being  developed  ahead  of  the  product.  Instead, 
DOD’s  approach  moves  toward  the  traditional  practice  of  concurrently 
developing  technologies  and  products,  which  often  raised  cost-benefit 
issues  as  a  result  of  cost  increases  and  schedule  delays  as  problems  are 
encountered  in  technology  development. 

To  demonstrate  DOD  s  commitment  to  acquisition  reform,  follow  best 
commercial  practices,  and  reduce  the  risk  of  future  cost  growth,  the 
program  office  should  focus  on  risk  reduction  efforts  by  maturing  critical 
technologies  prior  to  entering  engineering  and  manufacturing 
development,  and  it  should  be  allowed  to  do  so  without  the  penalty  of 
withdrawal  of  funding  support.  We  make  a  recommendation  that  the  Joint 
Strike  Fighter  program  office  adjust  its  currently  planned  engineering  and 
manufacturing  development  decision  date  of  March  2001  to  allow  adequate 
time  to  mature  critical  technologies  to  acceptable  maturity  levels  before 
awarding  the  engineering  and  manufacturing  development  contract. 


Background 


The  Joint  Strike  Fighter  is  the  centerpiece  of  DOD  s  tactical  aircraft 
modernization  plan,  which  includes  the  Air  Force  F-22  Raptor  and  the  Navy 
F/A-18  E/F  Super  Hornet.  The  program  is  structured  to  use  a  common 
production  line  to  produce  three  aircraft  variants  that  meet  conventional 
flight  requirements  for  the  U.S.  Air  Force,  short  take-off  and  vertical 
landing  characteristics  for  the  Marine  Corps,  and  carrier  operation 
suitability  needs  for  the  U.S.  Navy.  The  program  will  also  provide  aircraft  to 
the  British  Royal  Navy  and  Air  Force.  Table  1  shows  current  service  plans 
for  Joint  Strike  Fighter  use. 
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Table  1:  Military  Service  Needs  for  the  Joint  Strike  Fighter 

Service 

Quantity 

Planned  use 

Air  Force 

1,763 

Replacement  for  F-16  and  A-10;  complement  to  the  F-22 

Marine  Corps 

609 

Short  take-off  and  vertical  landing  aircraft  to  replace  AV-8B 
and  F/A-18  C/D 

Navy 

480 

Carrier-based,  multi-role,  first  day  of  war  survivable  strike 
fighter  to  complement  the  F/A-18  E/F 

Great  Britain 

150 

Short  take-off  and  vertical  landing  replacement  for  the  Sea 
Harrier  and  GR.7 

Source:  Joint  Strike  Fighter  program  office. 

DOD  expects  the  Air  Force  variant  of  the  Joint  Strike  Fighter  to  cost  about 
$28  million  per  unit;  the  Navy  variant  to  be  between  $31  million  and 
$38  million;  and  the  Marine  Corps  variant  to  cost  between  $30  million  and 
$35  million.3  Independent  estimates  are  not  so  optimistic.  For  example,  in 
congressional  hearings  held  in  March  1999,  the  Congressional  Budget 
Office  estimated  that  the  unit  cost  of  the  Joint  Strike  Fighter  could  be  as 
much  as  47  percent  to  51  percent  higher  than  expected,  depending  on 
which  variant  was  procured.  DOD  and  the  Congressional  Budget  Office 
estimates  vary  as  a  result  of  differing  estimating  techniques,  including 
estimating  the  cost  of  incorporating  stealth  technologies  into  the  Joint 
Strike  Fighter  design.  Figure  1  shows  planned  Joint  Strike  Fighter  aircraft 
designs  by  contractor. 


3  Expected  costs  include  the  cost  to  produce  the  basic  aircraft,  propulsion  system,  and 
avionics.  Costs  are  stated  in  fiscal  year  1994  dollars. 
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Figure  1:  Boeing  and  Lockheed  Martin  Joint  Strike  Fighter  Aircraft  Design  Concepts 
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Source:  Joint  Strike  Fighter  program  office. 


Acquisition  Strategy 
Designed  to  Achieve 
Affordability  Goals 
Through  Reduced 
Program  Risk 


The  focus  of  the  Joint  Strike  Fighter  Program  is  affordability — reducing  the 
development,  production,  and  ownership  costs  of  the  program  relative  to 
prior  fighter  aircraft  programs.  To  achieve  this  objective,  the  Joint  Strike 
Fighter  program  office  has  incorporated  various  DOD  and  commercial 
acquisition  initiatives  into  the  Joint  Strike  Fighter  acquisition  strategy. 
These  initiatives  include  modifying  the  traditional  weapons  acquisition 
cycle,  revising  the  requirements  determination  process,  and  advancing  the 
maturity  level  of  critical  technologies  so  they  represent  low  technical  risk 
before  the  engineering  and  manufacturing  contract  is  awarded.  The 
expectation  is  that  incorporating  these  initiatives  into  the  Joint  Strike 
Fighter  acquisition  strategy  will  avoid  cost  growth,  schedule  slippage,  and 
performance  shortfalls  that  have  been  experienced  in  other  weapons 
acquisition  programs. 


Acquisition  Strategy 
Modifies  Traditional 
Acquisition  Cycle 


The  Joint  Strike  Fighter  acquisition  strategy  modifies  the  cycle  that 
weapons  acquisition  programs  normally  follow.  For  example,  the  Joint 
Strike  Fighter  program  office  was  created  earlier  in  the  acquisition  cycle 
than  it  would  have  been  in  a  traditional  DOD  weapons  system  program. 
This  enabled  DOD  to  obtain  early  input  from  relevant  stakeholders 
(operators,  maintenance  personnel,  industry  representatives,  government 
engineers,  and  officials  within  the  intelligence  community)  to  ensure  that 
all  aspects  of  cost,  schedule,  performance,  and  resource  constraints  are 
included  in  decision-making.  In  addition,  the  program  has  encouraged 
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greater  involvement  from  the  international  community,  which  provides 
both  monetary  and  technical  support  (see  app.  I) . 

The  traditional  acquisition  cycle  has  also  been  modified  by  combining  the 
first  two  traditional  acquisition  phases — Concept  Exploration  and  Program 
Definition  and  Risk  Reduction-— into  one  phase,  known  as  Concept 
Demonstration.  Under  the  traditional  DOD  acquisition  cycle,  final 
performance  requirements  are  developed  early  in  the  Concept  Exploration 
phase  (see  fig.  2);  in  the  Joint  Strike  Fighter  program,  final  requirements 
are  determined  later  in  the  acquisition  cycle.  Program  officials  state  that 
this  modification  provides  the  flexibility  needed  to  conduct  cost  and 
performance  trade-offs  before  requirement  and  design  decisions  become 
final. 


Figure  2:  Comparison  of  Traditional  and  Joint  Strike  Fighter  Acquisition  Cycles 


Engineering  & 

Concept  Program  Definition  Manufacturing 

Exploration  &  Risk  Reduction  Development  Production 

Traditional  DOD 
acquisition 

a 

JSF  acquisition 

♦  ■ 

◄ 

1 

Concept  Demonstration  Phase 

^  Program  start 
|  Final  performance  requirements 

aJoint  Strike  Fighter. 

Source:  Joint  Strike  Fighter  program  office. 
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Acquisition  Strategy  Revises 
the  Requirements 
Determination  Process 


In  what  is  known  as  an  Operational  Requirements  Document,  DOD  defines 
aircraft  program  requirements  (such  as  range,  speed,  and  acceleration) 
very  early  in  the  acquisition  cycle  and  before  considering  the  aircraft 
design  necessary  to  meet  these  requirements.  For  the  Joint  Strike  Fighter, 
performance  requirements  are  instead  defined  in  five  phases;  in  each 
phase,  specific  aircraft  design  characteristics  are  determined  as 
performance  requirements  are  set.  During  each  phase,  performance 
requirements  are  considered  in  terms  of  the  established  cost  targets  so  that 
trade-offs  in  performance  capabilities  can  be  made  as  necessary.4  The  most 
significant  trade-off  to  date  in  the  Joint  Strike  Fighter  Program  was  the 
decision  to  equip  the  aircraft  with  one  versus  two  engines.  Identifying 
trade-offs  to  balance  requirements  for  the  affordability,  effectiveness,  and 
supportability5  of  the  aircraft  design  concept  represents  an  ongoing  effort 
between  the  government  and  the  two  competing  contractors. 


Acquisition  Strategy 
Designed  to  Reduce 
Technical  Risk 


According  to  the  Joint  Strike  Fighter  Single  Acquisition  Management  Plan, 
a  principal  objective  of  the  program  is  “to  demonstrate  to  a  low  level  of 
technical  risk  those  critical  technologies,  processes,  and  system 
characteristics  necessary  to  produce  an  affordable  family  of  strike  aircraft 
that  meets  all  participants’  needs.”  To  achieve  that  objective,  the  Joint 
Strike  Fighter  acquisition  strategy  is  designed  to  lower  technical  risk 
through  aircraft  flight  demonstrations  and  advanced  technology 
development  prior  to  awarding  the  engineering  and  manufacturing 
development  contract.  Specifically,  during  the  current  concept 
demonstration  phase,  DOD  requires  each  contractor  to 

•  demonstrate  specific  aircraft  capabilities  by  designing  and  building 
actual  flying  models, 

•  conduct  demonstrations  of  key  technologies  and  processes  unique  to 
each  contractor’s  aircraft  design,  and 

•  submit  their  preferred  Joint  Strike  Fighter  design  concept. 


4  This  process  is  referred  to  as  using  Cost  as  an  Independent  Variable. 

5  The  degree  to  which  system  design  characteristics  and  planned  logistics  resources, 
including  manpower,  meet  system  peacetime  readiness  and  wartime  utilization 
requirements. 
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Each  competing  contractor  is  required  to  design  and  build  two  aircraft  to 
demonstrate  the  following: 

•  commonality/modularity  to  validate  the  contractors’  ability  to  produce 
three  aircraft  variants  on  the  same  production  line; 

•  short  take-off  and  vertical  landing,  hover,  and  transition  to  forward 
flight  to  demonstrate  the  aircraft’s  ability  to  perform  specific  Marine 
Corps  and  Royal  Navy  missions;  and 

•  satisfactory  low  airspeed  carrier  approach  flying  and  handling  qualities 
to  demonstrate  the  aircraft’s  ability  to  perform  specific  Navy  missions. 

Each  contractor  will  also  be  required  to  submit  a  Preferred  Weapon  System 
Concept,  which  outlines  their  preferred  design  concept  for  developing  an 
affordable  Joint  Strike  Fighter  aircraft  to  meet  the  goals  specified  in  the 
final  requirements  document.  The  Preferred  Weapon  System  Concept  will 
include  results  from  the  flight  and  ground  demonstrations  and  will 
ultimately  be  used  by  DOD  to  select  the  winning  aircraft  design  and  to 
award  the  engineering  and  manufacturing  development  contract. 


Implementation  of 
Acquisition  Strategy 
Will  Not  Ensure  That 
Program  Objectives 
Are  Achieved 


Contrary  to  its  acquisition  strategy,  the  Joint  Strike  Fighter  Program  will 
not  enter  the  engineering  and  manufacturing  development  phase  with  low 
technical  risk.  The  aircraft  to  be  used  in  the  concept  demonstration  phase 
are  not  intended  to  demonstrate  all  of  the  Joint  Strike  Fighter  critical 
technologies.  Therefore,  these  technologies  will  be  at  low  levels  of 
technical  maturity  when  the  engineering  and  manufacturing  development 
contract  is  scheduled  to  be  awarded.  In  addition,  when  the  competing 
contractors  experienced  design  problems  and  cost  overruns,  DOD 
restructured  the  program  in  a  manner  that  is  moving  away  from  the  best 
commercial  practices  that  were  evident  in  the  original  strategy  and  is 
instead  moving  toward  traditional  practices  that  have  caused  problems  on 
other  programs. 


Critical  Technologies  Not 
Developed  to  Acceptable 
Levels 


The  aircraft  being  produced  during  the  concept  demonstration  phase  are 
not  intended  to  demonstrate  many  of  the  technologies  considered  critical 
for  achieving  Joint  Strike  Fighter  Program  cost  and  performance 
requirements,  such  as  those  for  integrated  avionics.  Instead,  many  of  these 
technologies  will  be  demonstrated  only  in  laboratory  or  ground-testing 
environments  and,  therefore,  will  be  at  low  levels  of  technical  maturity 
when  the  engineering  and  manufacturing  development  contract  is 
scheduled  to  be  awarded. 
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Measuring  Technology 
Readiness 


Matching  the  aircraft  requirements  and  the  maturity  of  technology  as  a 
program  begins  is  perhaps  the  most  important  determinant  of  a  program’s 
success.  Once  a  program  begins,  a  large,  fixed  investment  in  the  form  of 
human  capital,  facilities,  and  materials  is  sunk  into  the  program  and  any 
significant  changes  will  have  a  large,  rippling  effect  on  schedule  and  cost. 
In  the  case  of  critical  technologies,  beginning  an  acquisition  program  when 
the  technologies  are  at  a  low  level  of  development  increases  program  risk 
and  the  likelihood  of  schedule  delays,  which  increases  program  costs. 

At  our  request,  the  Joint  Strike  Fighter  program  office  identified  eight 
technology  areas  that  are  considered  critical  to  meeting  Joint  Strike  Fighter 
cost  and/or  performance  objectives.  These  technologies  address  areas  such 
as  avionics,  flight  systems,  manufacturing  and  producibility,  propulsion, 
supportability,  and  weapons  delivery  system.6  We  requested  the  program 
office  to  assign  maturity  levels  for  these  critical  technologies  using  a  tool 
referred  to  as  technology  readiness  levels  (TRLs) .  The  TRLs  were 
pioneered  by  the  National  Aeronautics  and  Space  Administration  and 
adopted  by  the  Air  Force  Research  Laboratory7  to  determine  the  readiness 
of  technologies  to  be  incorporated  into  a  weapon  or  other  type  of  system. 
The  Joint  Advanced  Strike  Technology  Program — from  which  the  Joint 
Strike  Fighter  Program  evolved — used  TRLs  to  assess  early  maturity  levels 
for  many  of  the  current  Joint  Strike  Fighter  technologies.  In  response  to 
our  prior  work,  DOD  has  agreed  that  TRLs  can  be  used  to  help  guide 
technology  maturation  and  transition  decisions.8  Detailed  descriptions  of 
technology  readiness  levels  can  be  found  in  appendix  II. 

In  conjunction  with  the  program  office  and  the  two  competing  contractors, 
we  determined  the  readiness  levels  of  critical  technologies  when  the  Joint 
Strike  Fighter  Program  was  started  in  1996.  That  assessment  showed  that 
when  the  Joint  Strike  Fighter  Program  entered  the  concept  demonstration 
phase,  most  of  the  critical  technologies  were  well  below  maturity  levels 


6  Due  to  the  current  Joint  Strike  Fighter  competition,  the  critical  technologies  are  not 
specified  so  as  not  to  associate  them  with  the  respective  contractors. 

7  The  Air  Force  Research  Laboratory  is  a  science  and  technology  organization  that  matures 
advanced  technologies  to  the  point  that  they  can  be  included  in  weapon  system  programs 
and  be  expected  to  perform  as  required.  The  Laboratory  uses  the  TRLs  to  assess  the 
maturity  of  the  technologies  before  they  are  handed  off  to  programs. 

8  Best  Practices:  Better  Management  of  Technology  Development  Can  Improve  Weapon 
System  Outcomes  (GAO/NSIAD-99-162,  July  30,  1999). 
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Maturity  Levels  of  Joint  Strike 
Fighter  Critical  Technologies 


considered  acceptable  for  low  risk  by  the  Air  Force  Research  Laboratory  or 
by  leading  commercial  firms. 

Readiness  levels  are  measured  on  a  scale  of  one  to  nine:  Studies  of  the 
basic  concept  have  a  readiness  level  of  one;  laboratory  demonstrations 
have  a  readiness  level  between  three  and  six;  and  technologies  that  have 
been  proven  through  integration  on  the  intended  product  have  a  readiness 
level  of  nine.  With  this  measurement  tool,  the  program  office  can  gauge  the 
likely  consequences  of  placing  various  technologies  at  a  given  maturity 
level  into  a  development  program  and  make  informed  choices  and  trade¬ 
offs  if  necessary  to  meet  program  goals. 

The  Air  Force  Research  Laboratory  considers  a  technology  readiness  level 
of  six  an  acceptable  risk  for  a  weapon  system  entering  the  program 
definition  stage,  the  point  at  which  DOD  typically  begins  its  weapon 
programs.  At  a  lower  level  of  technology  readiness,  the  technology’s  ability 
to  meet  the  intended  product’s  cost,  schedule,  and  performance 
requirements  is  uncertain.  Reaching  a  maturity  level  of  six  denotes  a 
significant  transition  point  for  technology  development— as  the  technology 
moves  from  component  testing  in  a  laboratory  environment  to 
demonstrating  a  model  or  prototype  in  a  relevant  environment.  Figure  3 
shows  the  maturity  levels  for  the  eight  critical  Joint  Strike  Fighter 
technologies  in  November  1996,  shortly  after  the  Joint  Strike  Fighter 
Program  was  started. 
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Figure  3:  Joint  Strike  Fighter  Critical  Technology  Readiness  Levels  at  Program  Start 


Critical  technology  areas 


Technology  1 
Technology  2 
Technology  3 
Technology  4 
Technology  5 
Technology  6 
Technology  7 
Technology  8 


0  12  3 


D  Program  start  (contract  award  - 1 1/96) 


Note:  Due  to  the  current  Joint  Strike  Fighter  competition,  the  critical  technologies  are  not  identified  so 
as  not  to  associate  them  with  the  respective  contractors. 

Source:  Joint  Strike  Fighter  program  office. 


As  shown  in  figure  3,  none  of  the  Joint  Strike  Fighter  critical  technologies 
had  achieved  the  desired  technology  readiness  level  of  six  by  program 
start.  Instead,  all  of  the  technologies  were  at  readiness  levels  of  two  or 
three,  which  means  that,  at  best,  analytical  and  laboratory  studies  had  been 
completed  or  very  early  components  had  been  developed. 


We  also  obtained  program  office  and  contractor  data  assessing  the 
expected  maturity  levels  for  the  Joint  Strike  Fighter  as  it  enters  engineering 
and  manufacturing  development  in  March  2001.  The  Air  Force  Research 
Laboratory  considers  TRL  7  as  acceptable  for  low  risk  when  entering  the 
engineering  and  manufacturing  development  stage.  This  maturity  level 
represents  an  advanced  prototype  of  each  of  the  critical  technologies 
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demonstrated  in  an  operational  environment,  such  as  on  a  flying  test  bed  or 
another  aircraft  similar  to  the  Joint  Strike  Fighter.  A  prototype  at  this  stage 
would  include  all  of  the  components  of  a  critical  technology  in  a 
configuration  that  is  very  close  to  the  size,  weight,  and  configurations  as 
that  expected  for  the  Joint  Strike  Fighter.  While  the  Joint  Strike  Fighter 
Program  has  seen  improvement  in  many  technology  areas  since  the 
program  started,  maturity  levels  have  not  improved  enough  to  indicate  a 
low-risk  transition  into  the  next  phase.  Figure  4  summarizes  this  data. 


Figure  4:  Joint  Strike  Fighter  Critical  Technology  Readiness  Levels  at  Program  Start  and  Projected  for  Entry  Into  Engineering 
and  Manufacturing  Development 


Acceptable  risk  for  program  start 


Acceptable  risk  for  entry  into 
engineering  and  manufacturing 
development  phase 


Technology  2 
Technology  3 
Technology  4 
Technology  5 
Technology  6 
Technology  7 
Technology  8 


Technology  readiness  level 

m  Program  start  (contract  award  - 11/96)  □  Current  (1 2/99)  ■  Engineering  and  manufacturing  development  start  (4/01 ) 


Note:  Due  to  the  current  Joint  Strike  Fighter  competition,  the  critical  technologies  are  not  identified  so 
as  not  to  associate  them  with  the  respective  contractors. 

Source:  Joint  Strike  Fighter  program  office. 
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As  shown  in  figure  4,  all  of  the  Joint  Strike  Fighter  critical  technologies  are 
expected  to  be  at  maturity  levels  lower  than  that  considered  acceptable  for 
low  risk  when  entering  engineering  and  manufacturing  development 
(TRL  7).  Six  of  the  technologies  will  still  be  below  the  level  of  maturity 
(TRL  6)  that  is  considered  low  risk  for  entering  the  demonstration  phase 
(program  start),  which  the  Joint  Strike  Fighter  Program  entered  over 
3  years  ago. 

Should  any  of  these  technologies  be  delayed  or,  worse  still,  not  be  available 
for  incorporation  into  the  final  Joint  Strike  Fighter  design,  the  impact  on 
the  program  would  be  dramatic.  For  example,  if  one  of  the  above  critical 
technologies  needed  to  be  replaced  with  its  planned  backup,  DOD  could 
expect  an  increase  of  several  billion  dollars  in  production  and  operation 
and  support  costs.9  The  backup  technology  would  also  significantly 
increase  aircraft  weight,  which  could  negatively  affect  aircraft 
performance.  The  currently  planned  technology  is  expected  to  be  TRL  5  at 
the  beginning  of  the  engineering  and  manufacturing  development  phase, 
which  indicates  that  substantial  technology  development  must  still  occur 
during  that  phase. 


DOD  Response  to 
Contractor  Cost  and 
Schedule  Problems  Is 
Inconsistent  With 
Acquisition  Strategy  Goals 


Recent  contractor  reports  indicate  that  both  competing  contractors  have 
cost  growth  and  schedule  concerns.  In  response  to  concerns  over  its  ability 
to  meet  program  requirements,  Boeing  has  redesigned  its  preferred  design 
configuration,  including  changing  the  wing  shape;  adding  a  horizontal  tail, 
which  lengthens  the  fuselage;  and  switching  from  a  forward  sweep  air 
intake  to  a  rearward  sweep.  According  to  a  Lockheed  Martin  program 
official,  the  company  underestimated  the  cost  of  producing  the  two 
demonstrator  aircraft.  In  addition,  Joint  Strike  Fighter  Program  documents 
suggest  that,  due  to  manufacturing  delays,  the  flight-test  schedule  for  both 
competing  contractors'  Marine  Corps  variant  could  be  at  risk. 


As  a  result  of  cost  concerns,  DOD  restructured  the  Joint  Strike  Fighter 
Program  to  allow  each  contractor  leeway  in  correcting  deficiencies.  For 
example,  the  Joint  Strike  Fighter  flight-test  program  has  been  decreased, 
which  will  reduce  the  data  available  for  final  proposal  evaluation.  Joint 
Strike  Fighter  Program  officials  stated  that  with  these  flight-test 
reductions,  only  the  minimum  acceptable  flight  quality  demonstrations  are 


9  Specific  details  cannot  be  provided  due  to  the  competitive  nature  of  the  Joint  Strike 
Fighter  Program. 
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Traditional  Approach  to 
Problems  Is  Underlying  Cause 
of  Increased  Risk 


expected.  The  number  of  preferred  weapon  system  design  updates  has  also 
been  reduced,  which  means  that  the  contractors  will  provide  DOD  with 
less  information  than  originally  planned  prior  to  the  submission  of  their 
proposals.  Finally,  DOD  has  eliminated  risk-reduction  efforts  and  delayed 
other  technology  demonstrations,  which  will  increase  the  program’s 
technical  risk  as  it  transitions  into  engineering  and  manufacturing 
development  (see  footnote  8) . 

DOD’s  traditional  approach  to  weapon  systems  acquisition  is  to  mature 
technology  at  the  same  time  the  product  is  being  developed.  This  approach 
differs  from  best  commercial  practices,  in  which  technology  is  developed 
separately  and  ahead  of  the  product.  Pressures  exerted  on  weapon  system 
programs  create  incentives  for  programs  to  include  immature  technologies 
that  may  offer  significant  performance  gains.  This  pressure  can  come  from 
users  who  demand  performance  improvements  that  necessitate  the 
application  of  unproven  technologies  to  stay  ahead  of  the  perceived  threat. 
Another  source  of  pressure  is  from  technologists,  who  see  a  new  weapon 
system  as  an  opportunity  to  apply  a  new  technology.  Also,  the  competition 
for  funds  can  encourage  performance  features — and  requisite 
technologies — that  can  distinguish  the  new  weapon  system  from 
competitors. 

Once  in  a  product  development  environment,  external  pressures  to  keep 
the  program  moving  (such  as  preserving  cost  and  schedule  estimates  to 
secure  budget  approval)  become  dominant.  For  example,  DOD  policies 
require  that  a  program  be  funded  in  the  current  year  and  that  funds  be 
made  available  over  the  next  6  years  in  the  DOD  planning  cycle.  If  a 
program  manager  decided  that  an  additional  year  was  needed  to  reach  the 
desired  level  of  technical  maturity  during  the  risk  reduction/concept 
demonstration  phase,  the  planned  start  of  the  engineering  and 
manufacturing  development  phase  could  be  delayed.  This  delay  could 
jeopardize  funding  for  that  phase,  thus  risking  the  funding  support  for  the 
entire  program.  Consequently,  the  program  manager  may  be  more  likely  to 
accept  the  risk  of  moving  forward  with  a  lower  level  of  technology  maturity 
rather  than  risk  losing  the  program.  That  decision  would  raise  cost-benefit 
issues  because  cost  increases  and  performance  compromises  would  likely 
occur. 
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Conclusions  The  J°int  Strike  Fighter  Program’s  acquisition  strategy  was  to  develop  and 

field  an  affordable  aircraft  that  will  meet  each  of  the  military  services’ 
requirements  goals.  However,  a  key  objective  of  the  acquisition  strategy — 
entering  into  engineering  and  manufacturing  development  with  low 
technical  risk — will  not  be  achieved  due  to  the  manner  in  which  the  Joint 
Strike  Fighter  program  office  is  implementing  the  acquisition  strategy  On 
its  current  schedule,  the  program  will  enter  the  engineering  and 
manufacturing  development  phase  without  having  reduced  to  an 
acceptable  level  the  technical  risk  of  technologies  that  the  program  office 
has  identified  as  critical  to  meeting  the  program’s  cost  and  requirement 
objectives.  This  approach  is  not  consistent  with  best  commercial  practices 
in  which  technologies  are  more  fully  developed  before  proceeding  into 
product  development.  It  is  also  not  consistent  with  DOD’s  planned 
approach  to  developing  the  Joint  Strike  Fighter.  Instead,  the  program 
office’s  revised  approach  is  consistent  with  DOD’s  traditional  approach  in 
weapon  system  programs  of  concurrently  developing  technologies  and 
products.  This  traditional  approach  has  often  raised  cost-benefit  issues  as  a 
result  of  cost  increases,  schedule  delays,  and  compromised  performance  as 
problems  arose  in  completing  technology  development. 

The  Joint  Strike  Fighter  Program  is  at  an  early  development  stage  and, 
therefore,  DOD  still  has  the  opportunity  to  both  demonstrate  its 
commitment  to  acquisition  reform  and  chart  a  course  to  avoid  the 
problems  that  often  befall  major  weapon  systems.  A  decision  to  allow  the 
Joint  Strike  Fighter  to  proceed  as  planned,  without  mature  critical 
technologies,  would  compromise  DOD’s  position  on  acquisition  reform,  set 
aside  best  commercial  practices,  and  would  perpetuate  conditions  that 
have  led  to  cost  growth  and  schedule  delays  in  many  prior  DOD  weapon 
system  acquisition  programs. 


Recommendation  To  demonstrate  DOD’s  commitment  to  acquisition  reform  and  to  reduce  the 

risk  of  future  cost  growth,  the  program  office  should  focus  on  risk 
reduction  efforts  by  maturing  critical  technologies  prior  to  entering 
engineering  and  manufacturing  development,  and  it  should  be  allowed  to 
do  so  without  the  penalty  of  withdrawal  of  funding  support.  Therefore,  we 
recommend  that  the  Secretary  of  Defense  direct  the  Joint  Strike  Fighter 
program  office  to  adjust  the  currently  planned  March  2001  engineering  and 
manufacturing  development  decision  date  to  allow  adequate  time  to 
mature  critical  technologies  to  acceptable  maturity  levels,  thereby  closing 
the  gap  between  technology  and  requirements,  before  awarding  the 
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engineering  and  manufacturing  development  contract.  Options  that  the 
program  should  consider  include 

•  delaying  the  selection  of  a  single  contractor  for  the  engineering  and 
manufacturing  phase  of  the  program  until  the  program’s  critical 
technologies  have  been  developed  to  an  acceptable  level  or 

•  selecting  a  single  contractor,  but  providing  the  time  and  funding  for 
additional  risk  reduction  and  technology  maturation  efforts,  so  that  this 
contractor  can  mature  critical  technologies  to  acceptable  levels  before  a 
decision  is  made  to  begin  engineering  and  manufacturing  development. 


Matters  for 

Congressional 

Consideration 


To  ensure  that  the  Joint  Strike  Fighter  Program  enters  the  engineering  and 
manufacturing  development  phase  with  low  technical  risk,  as  envisioned 
by  the  original  acquisition  strategy,  Congress  may  wish  to  consider 
requiring  the  Secretary  of  Defense  to  identify  which  of  the  eight  critical 
technologies  discussed  in  this  report  will  be  incorporated  on  the  Joint 
Strike  Fighter  and  certify  that  each  of  the  identified  technologies  has  been 
demonstrated  in  a  form  that  is  the  right  size,  weight,  and  configuration 
needed  for  the  Joint  Strike  Fighter  aircraft.  For  any  of  the  eight 
technologies  not  initially  included  on  the  Joint  Strike  Fighter,  the  Secretary 
of  Defense  should  develop  a  plan  showing  the  strategy  for  demonstrating 
these  technologies  in  the  right  size,  weight,  and  configuration;  showing  the 
approach  for  including  them  onto  the  Joint  Strike  Fighter;  and  the  cost 
impact  if  these  technologies  do  not  become  available  as  planned.  Congress 
may  also  want  to  consider  restricting  DOD  from  obligating  funds  made 
available  for  the  engineering  and  manufacturing  development  phase  of  the 
program  until  it  receives  this  information  from  DOD. 


Agency  Comments  and 
Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  the  Director  of  Strategic  and 
Tactical  Systems,  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics,  disagreed  with  our  conclusions  and 
recommendation.  DOD  stated  that  our  conclusion  was  based  on 
misinterpretation  of  the  use  of  technology  readiness  levels  to  determine  the 
readiness  of  the  critical  technologies  to  enter  engineering  and 
manufacturing  development.  In  addition,  DOD  stated  that  (1)  only  the 
maturity  of  the  technology,  not  its  integration  onto  the  Joint  Strike  Fighter, 
should  be  rated  to  determine  its  readiness  to  enter  engineering  and 
manufacturing  development;  (2)  our  use  of  technology  readiness  levels 
does  not  recognize  that  an  evolutionary  acquisition  approach  is  being 
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applied  to  the  Joint  Strike  Fighter  Program;  and  (3)  its  own  risk  reduction 
efforts  are  more  meaningful  than  using  technology  readiness  levels. 

Contrary  to  DOD’s  comments,  there  was  no  misinterpretation  by  us,  the 
contractors,  or  the  program  office  representatives  about  what  constituted  a 
readiness  level  of  seven — the  level  considered  necessary  for  acceptable 
risk  for  entering  engineering  and  manufacturing  development.  As  pointed 
out  in  this  report,  the  readiness  level  definitions  clearly  state  that  for  a 
technology  to  be  rated  at  a  level  seven,  it  must  be  demonstrated  using 
prototype  hardware,  such  as  a  complete  radar  subsystem  that  is  the  same 
size,  weight,  and  configuration  planned  for  the  Joint  Strike  Fighter  in  an 
operational  environment.  This  definition  was  developed  by  the  Air  Force 
Research  Laboratory  and  was  provided  to  the  contractors  prior  to  their 
scoring  of  the  critical  technologies,  and  we  discussed  them  at  length  during 
several  days  of  meetings  when  the  contractors,  with  DOD  program  office 
personnel  present,  assigned  the  readiness  levels  for  the  critical 
technologies.  During  those  discussions,  we  made  it  clear  that 
demonstrating  the  technology  in  a  relevant  environment  would  include 
demonstrating  the  technology  in  a  flying  test  bed  aircraft,  such  as  an  F-16 
or  some  other  existing  aircraft,  and  not  necessarily  on  an  actual  Joint 
Strike  Fighter  aircraft.  As  a  result  of  those  discussions,  there  was 
agreement  on  the  readiness  levels  assigned  to  each  of  the  critical 
technologies  discussed.  The  program  office  then  independently  scored  the 
critical  technologies.  The  program  office  scores,  which  are  those  used  in 
this  report,  were  consistent  with  the  contractors’  scores. 

DOD  stated  that  the  Joint  Strike  Fighter  Program  will  address  the 
integration  risk  of  the  critical  technologies  during,  rather  than  prior  to,  the 
engineering  and  manufacturing  development  phase.  As  indicated  in  our 
report,  we  agree  that  the  risk  of  integrating  a  subsystem-such  as  a  radar- 
onto  the  actual  Joint  Strike  Fighter  aircraft  is  an  activity  that  is  acceptable 
for  the  engineering  and  manufacturing  phase.  However,  we  do  not  agree 
that  integrating  various  components  of  a  subsystem — such  as  an  antenna, 
receiver,  transmitter,  and  processor  that  make  up  a  radar  subsystem — into 
a  configuration  that  can  be  inserted  into  the  Joint  Strike  Fighter  is  a  task  to 
be  left  for  the  engineering  and  manufacturing  development  phase.  In  that 
regard,  commercial  firms  have  told  us  that  a  key  part  of  technology 
development  is  getting  the  technology  into  the  right  size,  weight,  and 
configuration  needed  for  the  intended  product.  Once  this  has  been 
demonstrated,  the  technology  is  at  an  acceptable  level  for  engineering  and 
manufacturing  development,  where  the  emphasis  should  be  on  building  the 
actual  Joint  Strike  Fighter  aircraft.  In  separate  technical  comments  on  this 
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report,  DOD  acknowledged  that  the  highest  program  risks  will  come  from 
integrating  critical  technologies,  which  it  intends  to  do  during  engineering 
and  manufacturing  development.  DOD’s  approach,  of  developing 
technology  during  engineering  and  manufacturing  development,  represents 
a  fundamental  difference  between  best  commercial  practices  and  DOD 
practices  and  has  been  a  leading  reason  for  DOD’s  weapon  programs 
incurring  cost  growth,  schedule  slippage,  and  technical  problems. 

In  its  comments,  DOD  described  an  evolutionary  acquisition  approach 
being  applied  to  the  Joint  Strike  Fighter  Program  as  a  way  to  further 
mitigate  technology  and  program  risk.  This  approach  means  that  the  Joint 
Strike  Fighter  Program  will  use  time-phased  requirements  and  capabilities, 
and  only  those  technologies  and  capabilities  that  are  mature  and  at  low  risk 
will  be  installed  on  the  first  Joint  Strike  Fighter  aircraft  that  are  produced. 
Cost  and  requirement  trade-offs  will  be  considered  so  that  technologies 
will  not  be  included  on  the  Joint  Strike  Fighter  until  they  have 
demonstrated  an  acceptable  maturity  level.  We  are  concerned  that  since 
many  of  the  technologies  assessed  at  low  maturity  levels  in  our  report  are 
critical  to  obtaining  an  affordable  aircraft — a  primary  objective  for  both  the 
Joint  Strike  Fighter  and  DOD’s  overall  tactical  aircraft  modernization 
plan — their  absence  from  the  Joint  Strike  Fighter  design  could  result  in 
much  higher  development,  production,  and  support  costs.  The  impact  of 
developing  and  producing  Joint  Strike  Fighter  aircraft  without  these 
technologies  must  be  considered,  otherwise,  DOD  could  find  itself 
committed  to  a  program  dependent  on  future  maturation  of  these 
technologies  in  order  to  meet  program  cost  objectives.  This  would  be 
typical  of  DOD’s  historical  approach  to  developing  weapon  systems,  and  it 
indicates  that  DOD  is  willing  to  assume  greater  risk  during  engineering  and 
manufacturing  development  than  was  envisioned  in  the  Joint  Strike  Fighter 
Program’s  original  acquisition  strategy. 

Finally,  DOD  stated  that  its  own  risk  reduction  methodology-which  include 
risk  mitigation  plans  and  engineering  judgment-is  a  more  meaningful 
measure  of  risk  versus  the  use  of  technology  readiness  levels.  We  do  not 
share  DOD’s  confidence  in  this  regard.  The  objective  of  technology 
readiness  levels  is  to  make  decisions  based  on  actual  demonstrations  that 
technologies  will  work  as  needed  for  the  intended  product  as  opposed  to 
engineering  judgment,  which  is  subjective  and  open  to  interpretation.  Our 
prior  work  has  shown  that  in  place  of  risk  mitigation  plans  and  engineering 
judgment,  no  matter  how  well  intentioned,  using  technology  readiness 
levels  results  in  a  straightforward,  objective,  and  quantifiable  process  for 
determining  a  technology’s  readiness  for  proceeding  into  the  engineering 
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and  manufacturing  development  phase  of  a  program.  DOD  has  recognized 
the  value  of  the  technology  readiness  level  process  as  indicated  by  its 
intention  to  include  the  process  in  the  revised  acquisition  guidelines — 
commonly  referred  to  as  the  5000  acquisition  series. 

In  summary,  the  information  provided  by  DOD  in  its  comments  on  our  draft 
report  does  not  change  our  position  that,  as  currently  structured,  the  Joint 
Strike  Fighter  Program  will  move  into  the  engineering  and  manufacturing 
development  phase  with  unacceptable  risk  levels  for  many  critical 
technologies.  Therefore,  we  have  not  changed  the  recommendation  that 
was  in  our  draft  report  but  we  have  added  some  matters  for  Congress  to 
consider.  In  essence,  we  suggest  that  Congress  consider  restricting  DOD 
from  obligating  funds  for  the  engineering  and  manufacturing  development 
phase  of  the  program  until  the  Secretary  of  Defense  certifies  that  the 
program’s  critical  technologies  have  been  demonstrated  to  acceptable 
levels  of  maturity. 

The  full  text  of  DOD’s  comments  are  included  in  appendix  III.  DOD  also 
provided  separate  technical  comments  that  we  have  incorporated  into  the 
report  as  appropriate. 


Scope  and 
Methodology 


To  determine  the  acquisition  strategy  for  meeting  affordability  goals 
through  reduced  risk  prior  to  awarding  the  engineering  and  manufacturing 
development  contract,  we  reviewed  program  documentation  on  acquisition 
strategy  and  acquisition  reform  initiatives  and  discussed  these  materials 
with  DOD  and  program  officials.  These  materials  include  data  such  as  the 
Joint  Strike  Fighter  Single  Acquisition  Management  Plan,  the  Interim  Test 
and  Evaluation  Master  Plan,  Joint  Interim  Requirements  Document,  Key 
Performance  Parameters  in  the  draft  Joint  Operational  Requirements 
Document,  and  Cost  as  an  Independent  Variable  documentation. 

To  determine  whether  the  program  office  is  implementing  the  Joint  Strike 
Fighter  acquisition  strategy  in  a  manner  that  will  reduce  risk  and  meet  Joint 
Strike  Fighter  cost,  schedule,  and  performance  goals,  we  collected  data 
and  interviewed  officials  at  various  DOD  locations,  the  Joint  Strike  Fighter 
program  office  (Arlington,  Virginia),  and  cognizant  Navy,  Marine  Corps,  and 
Air  Force  requirements  organizations.  We  interviewed  officials  and 
collected  contractor  data  from  Lockheed  Martin  Tactical  Aircraft  Systems, 
Fort  Worth,  Texas,  and  Palmdale,  California;  the  Boeing  Company,  Seattle, 
Washington,  and  Palmdale,  California;  General  Electric,  Cincinnati,  Ohio; 
and  Pratt  &  Whitney,  West  Palm  Beach,  Florida. 
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To  assess  the  maturity  levels  of  key  technologies,  we  used  information 
provided  by  the  Joint  Strike  Fighter  program  office  and  contractors  and 
used  the  technology  readiness  level  tool  developed  by  the  National 
Aeronautics  and  Space  Administration.  On  separate  visits  to  the 
contractors,  with  DOD  program  office  personnel  present,  we  provided 
relevant  technology  managers  the  TRL  scoring  sheet  found  in  appendix  II 
of  this  report.  After  significant  discussion,  and  additional  TRL  information, 
we  asked  these  managers  to  score  those  technologies  they  considered 
critical  to  enable  their  Joint  Strike  Fighter  design  to  meet  DOD 
requirements  for  the  aircraft.  Upon  reviewing  these  scores  with  the 
program  office  and  in  order  to  gain  an  overall  Joint  Strike  Fighter  Program 
perspective  on  technical  maturity,  the  Joint  Strike  Fighter  office  agreed  to 
provide  us  with  TRL  scores  for  the  eight  technologies  they  considered 
critical  for  meeting  program  cost  and  performance  requirements.  Those 
scores  are  presented  in  this  report. 

We  conducted  our  review  from  September  1998  through  January  2000  in 
accordance  with  generally  accepted  government  auditing  standards. 


Unless  you  publicly  announce  its  contents  earlier,  we  plan  no  further 
distribution  of  this  report  until  30  days  from  its  issue  date.  At  that  time,  we 
will  send  copies  to  the  congressional  defense  committees;  the  Honorable 
William  S.  Cohen,  Secretary  of  Defense;  the  Honorable  F.  Whitten  Peters, 
Secretary  of  the  Air  Force;  the  Honorable  Richard  Danzig,  Secretary  of  the 
Navy;  General  James  L.  Jones,  Commandant,  U.S.  Marine  Corps;  and  the 
Honorable  Jacob  J.  Lew,  Director,  Office  of  Management  and  Budget.  We 
will  also  make  copies  available  to  other  interested  parties  on  request. 
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Please  contact  me  at  (202)  512-4841  if  you  or  your  staff  have  any  questions 
concerning  this  report.  Key  contributors  to  this  assignment  were  Steve 
Kuhta,  Brian  Mullins,  Delores  Cohen,  and  Matt  Lea. 


Sincerely  yours, 


Louis  J.  Rodrigues 

Director,  Defense  Acquisitions  Issues 
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International  Participation  in  the  Joint  Strike 
Fighter  Program 


Full  collaborative  partner 

•  full  access  to  program  data  and  structure,  including  representative 
personnel  within  the  program  office 

•  ability  to  influence  requirement  definition  and  performance 
characteristics 

•  Great  Britain  is  the  only  participant  in  this  category,  committing 
$200  million  for  the  concept  demonstration  phase 

Associate  partner 

•  limited  access  to  data  and  limited  requirements  influence 

•  representative  personnel  resident  within  the  Joint  Strike  Fighter 
program  office 

•  all  aspects  of  participation  are  negotiated  with  the  Joint  Strike  Fighter 
program  office 

•  Denmark,  the  Netherlands,  and  Norway  formed  a  group  and  committed 
a  total  of  $30  million  to  participate  at  this  level 

Informed  customer 

•  limited  access  to  program  information  and  representation  within  the 
program  office  is  negotiable 

•  no  influence  on  requirements 

•  Canada  and  Italy  have  committed  $10  million  each  to  participate  at  this 
level 

Major  participants 

•  recently  created  category;  also  referred  to  as  Foreign  Military  Sales 
participation  or  Fee  for  Service 

•  negotiate  directly  with  the  program  office  for  specific  Joint  Strike 
Fighter  Program  information  (e.g.,  Cost  and  Operational  Performance 
Trade  processes  and  modeling  and  simulation  studies) 

•  no  representative  personnel  resident  within  the  Joint  Strike  Fighter 
program  office 

•  Singapore,  Turkey,  and  Israel  are  currently  participating  at  this  level 
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Technology  Readiness  Levels  and  Their 
Definitions 


Technology  readiness  level 

Description 

1.  Basic  principles  observed  and  reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to  be  translated  into 
applied  research  and  development.  Examples  might  include  paper  studies  of  a  technology’s 
basic  properties 

2.  Technology  concept  and/or  application 
formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical  applications  can  be 
invented.  The  application  is  speculative  and  there  is  no  proof  or  detailed  analysis  to  support 
the  assumption.  Examples  are  still  limited  to  paper  studies. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  initiated.  This  includes  analytical  studies  and  laboratory 
studies  to  physically  validate  analytical  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or  representative. 

4.  Component  and/or  breadboard  validation 
in  laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that  the  pieces  will  work 
together.  This  is  relatively  “low  fidelity"  compared  to  the  eventual  system.  Examples  include 
integration  of  “ad  hoc”  hardware  in  a  laboratory. 

5.  Component  and/or  breadboard  validation 
in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic  technological 
components  are  integrated  with  reasonably  realistic  supporting  elements  so  that  the 
technology  can  be  tested  in  a  simulated  environment.  Examples  include  “high  fidelity" 
laboratory  integration  of  components. 

6.  System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment. 

Representative  model  or  prototype  system,  which  is  well  beyond  the  breadboard  tested  for 
technology  readiness  level  (TRL)  5,  is  tested  in  a  relevant  environment.  Represents  a  major 
step  up  in  a  technology’s  demonstrated  readiness.  Examples  include  testing  a  prototype  in 
a  high  fidelity  laboratory  environment  or  in  simulated  operational  environment. 

7.  System  prototype  demonstration  in  an 
operational  environment. 

Prototype  near  or  at  planned  operational  system.  Represents  a  major  step  up  from  TRL  6, 
requiring  the  demonstration  of  an  actual  system  prototype  in  an  operational  environment, 
such  as  in  an  aircraft,  vehicle,  or  space.  Examples  include  testing  the  prototype  in  a 
test  bed  aircraft. 

8.  Actual  system  completed  and  “flight 
qualified”  through  test  and  demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under  expected  conditions.  In 
almost  all  cases,  this  TRL  represents  the  end  of  true  system  development.  Examples 
include  developmental  test  and  evaluation  of  the  system  in  its  intended  weapon  systems  to 
determine  if  it  meets  design  specifications. 

9.  Actual  system  “flight  proven”  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mission  conditions,  such  as 
those  encountered  in  operational  test  and  evaluation.  In  almost  all  cases,  this  is  the  end  of 
the  last  “bug  fixing"  aspects  of  true  system  development.  Examples  include  using  the 
system  under  operational  mission  conditions. 

Source:  Best  Practices:  Better  Management  of  Technology  Development  Can  Improve  Weapon 
System  Outcomes  (GAO/NS  I  AD-99-1 62,  July  30,  1999). 
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Comments  From  the  Department  of  Defense 


Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


See  comment  1. 


ACQUISITION  AND 
TECHNOLOGY 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 


1?  APR  2000 


Mr.  Louis  J.  Rodrigues 
Director,  Defense  Acquisitions  Issues 
National  Security  and  International 
Affairs  Division 

U.  S.  General  Accounting  Office 
Washington,  DC  20548 


Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  General  Accounting 
Office  (GAO)  draft  report,  “JOINT  STRIKE  FIGHTER  ACQUISITION:  Development 
Schedule  Should  be  Changed  to  Allow  Risks  to  be  Reduced,”  dated  February  24, 2000 
(GAO  Code  707386/OSD  Case  1952).  The  Department  of  Defense  does  not  concur  with 
the  recommendation  in  the  draft  report  (see  enclosure). 

The  draft  report  states  that  the  implementation  of  the  Joint  Strike  Fighter 
Program’s  acquisition  strategy  will  not  ensure  that  the  program  will  enter  the  Engineering 
and  Manufacturing  Development  (EMD)  phase  with  low  technical  risk.  The  Department 
does  not  agree  with  this  conclusion,  which  is  based  on  misinterpretation  of  a  process  for 
determining  the  readiness  of  technologies  for  incorporation  into  major  systems. 

The  Joint  Strike  Fighter  Program  Office,  in  conjunction  with  each  competing 
contractor,  has  identified  critical  technologies,  processes,  and  system  characteristics 
required  for  the  program,  tailored  to  each  contractor’s  design.  Risks  have  been  identified, 
baselined,  and  tracked  to  document  the  specific  events  required  to  reduce  the  risk  of  these 
critical  technologies,  processes,  and  system  characteristics  to  a  low  level  prior  to  EMD 
initiation.  Implementation  of  this  acquisition  strategy  has  not  changed  significantly  since 
the  program  entered  the  Program  Definition  and  Risk  Reduction  phase  in  1996. 

JSF’s  Single  Acquisition  Management  Plan,  approved  by  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  on  November  15,  1996,  clearly  states  that  the 
goal  of  the  JSF  technology  maturation  program  is  to  evolve  the  most  promising  leading- 
edge  technologies  to  a  low  level  of  risk  prior  to  integration  during  the  JSF  EMD  phase. 
The  GAO  ground  rules  for  scoring  the  Technology  Readiness  Levels  (TRLs)  included  the 
risk  of  integrating  the  technology  onto  the  JSF  platform.  The  JSF  Program  Office  used 
those  ground  rules  to  arrive  at  the  ratings  contained  in  the  draft  report,  which  GAO  used 
as  a  basis  for  its  conclusion.  Upon  review  and  discussion  with  other  users  of  TRLs,  the 
Program  Office  determined  that  only  the  maturity  of  the  technology,  not  its  integration, 
should  be  rated  to  determine  its  readiness  to  enter  EMD.  When  this  is  done  (see 
enclosure),  one  finds  that  technology  risk  is  expected  to  be  at  an  acceptable  level  at  EMD 
start. 
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See  comment  2. 


In  a  response  to  a  previous  GAO  report,  the  Department  agreed  that  it  is  desirable 
to  mature  technologies  to  TRL  7  prior  to  entering  EMD.  However,  that  response  also 
noted:  ‘The  Department  agrees  that  TRLs  are  an  important  input  and  are  necessary,  but 
adds  that  they  are  not  sufficient  alone  to  decide  when  and  where  to  insert  new 
technologies  into  weapon  system  programs.” 

In  addition,  an  evolutionary  acquisition  approach,  which  includes  time-phased 
requirements  and  capabilities,  is  being  applied  to  the  JSF  program.  This  further  mitigates 
both  technology  and  program  risks  by  allowing  technologies  to  be  inserted  into  the 
program  as  they  mature.  This  also  allows  the  JSF  program  to  insert  the  basic 
functionality  of  any  of  these  technologies  and  then  to  grow  that  functionality  over  time. 
What’s  crucial  is  to  risk-manage  the  technologies  during  EMD  and  to  have  fail-back 
plans  in  place  in  the  event  some  fail  to  mature. 

Technical  comments  for  accuracy  and  clarification  have  been  provided  separately. 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 

Sincerely, 


George  R.  Schneiter 
Director 

Strategic  and  Tactical  Systems 

Enclosure 
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GAO  DRAFT  REPORT,  Dated  February  24.  2000 
(GAO  Code  707386/OSD  Case  19521 

“JOINT  STRIKE  FIGHTER  ACQUISITION:  Development  Schedule  Should  Be 
Changed  To  Allow  Risks  To  Be  Reduced” 

POD  COMMENTS  ON  THE  GAO  RECOMMENDATION 

•  RECOMMENDATION :  The  GAO  recommended  that  the  Secretary  of  Defense  direct 
the  JSF  program  office  to  adjust  the  currently  planned  March  2001  Engineering  and 
Manufacturing  Development  decision  date  to  allow  adequate  time  for  critical 
technologies  to  reach  acceptable  maturity  levels,  thereby  closing  the  gap  between 
technology  and  requirements,  before  awarding  the  Engineering  and  Manufacturing 
Development  contract. 

•  POD  RESPONSE:  Nonconcur.  JSF  critical  technologies,  processes,  and  system 
characteristics  will  be  low  risk  for  EMD  entry  in  March  2001  if  the  planned  program 
activities  are  carried  out  by  then. 

•  JSF  reduces  risks  of  individual  critical  technologies  to  low  risk  prior  to  EMD. 

•  JSF  addresses  integration  risk  of  those  technologies  on  a  JSF  platform  in  EMD. 

•  JSF  has  been  consistent  in  implementation  of  its  acquisition  strategy  with  regard  to 
risk  management  from  program  inception.  Risks  have  been  identified,  baselined,  and 
tracked,  documenting  the  specific  events  required  to  reduce  the  risk  of  these  critical 
technologies,  processes,  and  system  characteristics  to  low  prior  to  EMD  start. 

•  The  GAO’s  supplied  ground  rules  for  scoring  the  Technology  Readiness  Levels 
(TRLs)  included  technology  maturation  and  the  integration  risk.  The  JSF  Program 
Office  also  re-scored  the  technology  areas  using  just  technology  maturation  risk. 

Both  of  these  results  are  shown  in  the  attached  Table.  When  just  the  technology 
maturation  risks  are  addressed,  the  eight  highest-risk  technology  areas  are  rated  as 
either  TRL  6  or  7  at  the  start  of  EMD,  which  is  acceptable  for  entry  into  EMD. 

•  In  a  DoD  response  to  the  GAO  Report:  ’’BEST  PRACTICES:  Better  Management  of 
Technology  Development  Can  Improve  Weapon  System  Outcomes”  (GAO/NSIAD- 
99-162),  GAO,  July  1999,  the  Department  states  the  following:  “The  Department 
concurs  with  the  GAO  that  the  weapon  system  program  manager  should  assure  that 
technology  is  matured  to  TRL  7  before  insertion  into  a  new  system.”  The 
Department  has  also  stated  in  that  response:  “  ..that  TRLs  are  an  important  input  and 
are  necessary,  but  adds  that  they  are  not  sufficient  alone  to  decide  when  and  where  to 
insert  new  technologies  into  weapon  system  programs.” 

•  An  evolutionary  acquisition  approach,  which  includes  time-phased  requirements  and 
capabilities,  is  being  applied  to  the  JSF  program.  This  further  mitigates  both 
technology  and  program  risks  by  allowing  technologies  to  be  inserted  into  the 
program  as  they  mature.  This  also  allows  the  JSF  program  to  insert  the  basic 
functionality  of  any  of  these  technologies  and  then  to  grow  that  functionality  over 
time. 
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ESTIMATED  TRL  RATINGS  FOR  EMD  START  2001 


AREA 

GAO  DRAFT  REPORT 
(Technology  Maturation 
and  Integration  Risk) 

RATIONALE 

JSF  PROGRAM  OFFICE 
RE-SCORING 
(Technology  Maturation 
Alone) 

RATIONALE* 

Technology  1 

6 

Not  flown  on  JSF  in  a  JSF 
environment. 

7 

Flying  on  Concept 
Demonstrator  Aircraft 
(CD  A). 

Technology  2 

4 

One  portion  not  flown  in  JSF 
application. 

6 

Technology  already  flown 
in  aircraft,  helicopter, 
spacecraft,  etc. 

4 

No  equivalent  integrated 
support  in  current  US 
military  service. 

6 

kbh 

5 

Not  flown  on  JSF  in  a  JSF 
environment. 

7 

Flying  on  F-16. 

Technology  5 

|  5 

Not  flown  on  JSF  in  a  JSF 
environment. 

6 

Technology  has  been  flown 
on  JSF  flying  test  bed. 

Technology  6 

5 

Not  flown  on  JSF  in  a  JSF 
environment. 

7 

Technology  has  been  flown 
on  JSF  flying  test  bed. 

Technology  7 

4 

6 

Fusion  technology  has 
been  flown  on  JSF  flying 
test  bed. 

Technology  8 

6 

Limited  demonstration  on 
Concept  Demonstrator 
Aircraft. 

7 

Flying  on  CDA  as  well  as 
F-22,  F/A-18E/F, 
Eurofighter,  etc. 

♦Further  details,  which  are  proprietary,  are  available. 


Attachment:  a/s 
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The  following  are  GAO’s  comments  on  the  Department  of  Defense’s  (DOD) 
letter  dated  April  17,  2000. 


GAO  Comments 


DOD  provided  reevaluated  technology  readiness  levels  to  indicate  that 
if  the  risk  of  integrating  critical  technologies  was  not  considered,  then 
the  technology  risk  is  expected  to  be  at  an  appropriate  level. 
Notwithstanding  our  disagreement  over  whether  integration  risks 
should  be  considered  in  this  assessment,  we  believe  DOD  s  standards 
are  below  minimum  acceptable  levels.  Only  half  of  the  critical 
technologies  are  projected  to  be  at  readiness  level  7  while  the  other  half 
will  still  be  at  readiness  level  6.  We  disagree  with  DOD  that  readiness 
level  6  is  acceptable  for  low-risk  entry  into  engineering  and 
manufacturing  development.  Leading  commercial  firms  typically  insist 
on  a  readiness  level  8-a  higher  standard  than  that  used  by  the  Air  Force 
Research  Laboratory-before  a  technology  can  be  included  on  a 
product. 


2.  DOD  noted  that  in  addition  to  technology  readiness  levels,  other 

considerations  were  necessary  to  decide  when  and  where  to  insert  new 
technologies  into  weapon  system  programs.  While  DOD  did  not 
elaborate  on  what  other  considerations  were  applicable  to  the  Joint 
Strike  Fighter  Program,  its  similar  response  to  our  prior  work  involving 
technology  readiness  levels  referred  to  such  considerations  as  the 
increasing  projected  life  for  new  weapon  systems,  total  ownership 
costs,  and  urgency  based  upon  threat  assessments.  Many  of  the 
technologies  presented  in  this  current  report  directly  address  these 
other  considerations  and  are  critical  to  the  success  of  the  Joint  Strike 
Fighter  Program.  Therefore,  using  TRLs  in  the  Joint  Strike  Fighter 
Program  not  only  allows  DOD  to  manage  performance  risk,  but  also  to 
address  those  technologies  critical  to  meeting  these  other 
considerations.  With  regard  to  risk  assessment,  leading  commercial 
firms  who  have  adopted  knowledge-based  risk  assessment  approaches, 
such  as  TRLs,  have  produced  results  such  as  more  technically 
advanced,  higher  quality  products,  in  less  time  and  at  a  lower  cost  than 
their  predecessors. 


(707386) 
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U.S.  General  Accounting  Office  / 

P.O.  Box  37050  / 

Washington,  20013  / 

Orders  by  visitmg:  / 

Room  1100  \  / 

700  4th  St.  NW  (corner  of  4th ^nd  G  Sts.  NW) 

U.S.  General  Accounting  Office 
Washington,  DC  \  / 

Orders  by  phone: 

(202)  512-6000  /\ 

fax:  (202)  512-6061  /  \ 

TDD  (202)  512-2537  /  \ 

Each  day,  GAO  issues  a  list  of  neWy  available  reports  and 
testimony.  To  receive  facsimile  copies  of  the  daily  list  or  any  list 
from  the  past  30  days,  please  call  (2*(2)  512-6000  using  a  touchtone 
phone.  A  recorded  menu  will  provide  Information  on  how  to  obtain 
these  lists.  /  \ 

Orders  by  Internet:  \ 

For  information  on  how  to  access  GAO  reports  on  the  Internet, 
send  an  e-mail  message  with  “info”  in  the  bo^v  to: 


info@\ 


r.gao.gov 


or  visij/GAO’s  World  Wide  Web  home  page  at: 
http://www.gao.gov 


To  Report  Fraud, 
Waste,  or  Abuse  in 
Federal  Programs 


Contact  one: 

•  Web  site:  http://www.gao.gov/fraudnet/fraudnet.htm 

•  e-mail:  fraudnet@gao.gov 

•  1-800-424-5454  (automated  answering  system) 
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